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1  -  INTRODUCTION 


1  . 1  GENERAL 

In  selecting  an  efficient  context-free  parser  for  inclusion 
in  a  translator  writing  system  (TWS)  [2,8],  two  general  classes 
of  bottom-up  parsers  are  available:  those  based  on  precedence 
analysis  [3,4]  and  those  related  to  LR  (k)  analysis  [1,11]. 


Precedence 

techniques  have 

been  extensively 

used 

in 

practical  syst 

ems  (e.g.  XPL) 

[8,9]  being  both 

fast 

a  nd 

efficient.  LR (k)  techniques,  however,  although  theoretically 
efficient  and  fast  have  been  in  practice  not  available.  Either 
the  basic  parser  was  too  large  or  too  slow  (when  not  table 
driven)  or  the  tables  were  too  large  (Korenjak  [7]  suggested  a 
practical  though  had  hoc  technique  for  decreasing  this  size) , 
often  reflecting  inefficiencies  unrelated  to  the  language  itself 
(e.g.  the  re-reading  of  BNF  non-ter mi nals) . 

However,  DeRemer  [1]  in  the  interest  of  efficiency 
suggested  a  more  practical  approach.  He  expanded  on  the  idea  of 
converting  a  grammar  to  an  equivalent  FSM  which  in  turn  could  be 
transformed  to  an  equivalent  DPDA.  Knuth's  LR(k)  constructor 
algorithm  [6]  was  available  to  produce  these  FSfi*s  but  direct 
application  generally  resulted  in  very  large  machines.  An 
alternate  approach  was  to  use  Knuth*s  LR  (0)  constructor 
algorithm,  or  equivalently,  Earley's  [11]  interpretation  of  the 
algorithm,  to  generate  an  FSM  which  was  inadequate  in  some  sense 


- 


2 


but  which  resulted  in  a  m 
inadequacies  could  than  ba 

The  calculations  requ 
directly  as  the  complex 
direct  reflection  of  the  g 
a  hierarchy  of  LH (k)  g 
complexity  by: 


uch 

more 

modest 

roach 

rera 

oved 

by  the 

addi 

ired 

to 

resolve 

the 

ity  of  the  machine  ( 
ramraar) .  Accordingly 
rammars  given  in 


ine.  The  generated 
tion  of  lookahead- 

inadequacies  vary 
which  in  turn  is  a 
,  DeReraer  defined 
ascending  order  of 


1)  LR  (0) 

2)  SLR  (k) 

3)  LALR(k) 
h)  LR(k) 


LR (0)  grammars  have  no  inad 
grammars  have  FSM's  which  ca 
succeeding  each  inadequate  state 
grammars  have  FSM’s  which  can  o 
both  preceding  and  succeeding  ea 
and  right  context) .  Grammars 
however ,  are  such  that  the  inf or 
state  has  been  effectively  com 
states)  to  the  point  where  it  is 
inadequacy.  A  solution  is  to 
splitting  and  to  subsequently  re 
LALR (k)  technique. 


equate  states  to  resolve.  SLR  (k) 
n  be  resolved  with  information 
(i.e.  right  context).  LALR  (k) 
nly  be  resolved  with  information 
ch  inadequate  state  (i.e.  left 
which  are  LR  (k)  but  not  LALR(k), 
mation  preceding  the  inadequate 
pa c ted  (via  merging  of  identical 
insufficient  to  resolve  the 
undo  the  merging  via  state 
solve  the  inadequacy  using  the 


Although  DeRemer's  construction  is  termed  "practical”,  no 
working  parser  generator  has  previously  been  available  which 
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could  furnish  relevant  statist 

The  DeRemer  approach  was 
heavy  optimizations  in  both 
resulting  generated  parsers 
competitive  with  precedence  pa 


ICS. 


therefore  implemented  with 

very 

the 

space  and  time  domains 

.  The 

are 

in  fact  efficient 

and 

rsers 

in  both  domains. 

. 
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1.2  Notation,  Preliminary  Definitions 

It  is  assumed  that  the  reader  is  familiar  with  the 

properties  of  symbols  ,  strings  of  symbols,  finite _ state 

machines  (FSM’s),  and  deterministic  pushdown  automata  (DPDA ' s) , 

A  context-free  (CF)  grammar  is  a  quadruple  (V  ,V  ,G,P) 

N 

where  is  a  finite  set  of  symbols  called  terminals  ,  VN  is  a 

finite  set  of  symbols  distinct  from  those  in  V  called 

nonterminals  ,  G  is  a  distinguished  member  of  called  the  goal 
symbol  and  P  is  a  finite  set  of  pairs  called  productions  .  Each 
production  is  written  A  ->  w  and  has  a  left_part  A  in  V  and 

ri<jht_part  w  in  V*  where  V  =  V^UV^.  V+  denotes  the  set  of  all 

strings  composed  of  symbols  in  V,  including  the  empty  string. 

Without  loss  of  generality,  the  productions  are  arbitrarily 
numbered  from  1  to  N.  The  special  symbols  (#  (1 ),#  (2 ),...,#  ( N )  } 
called  (apply  symbols)  are  associated  one-for-one  with 

each  production.  The  P  th  production  is  conceptually  written 
with  #  (P)  as  the  rightmost  symbol  of  the  production 
(A  ->  w  #  (P)  )  . 

In  particular  the  #-symbol  for  production  P  is  referred  to 
as  the  #  (P)  symbol  or  as  #  (P)  . 

For  a  production  A  ->  w ,  an  immediate _ derivation  of  a 

string  a  =  bwc  from  another  a*  =  bAc  is  written  a'  ->  a.  We  say 
a  is  immediately  derivable  from  a1  via  application  of  the 
production  A  ->  w  to  a  particular  occurrence  of  A  in  a* .  A 


Jjwl 
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derivation 

derivation 
strings  ac,a 
for  m  >-  0. 
which  for  i= 

a  .  via 

t-1 

nonterminal 
chosen  as  th 

In  gene 
that  string 
string  a  is 
equivalently 
derivation  o 
Pdl  si  ng._  Th 
and  a  parsin 


is  the  transitive  completion  of 


an 


and  is  written  a*  ->  a.  It  implies  that  t 

a^  such  that  a*  ->  a  ->  a  ->  . . 

±  tn  o  i 

A  right _ derivation  ,  written  a*  ->  a 

1,2,...,m,  each  a.  is  immediately  deri 
application  of  a  production  to  the 

in  a*  .  For  LR  (k)  grammars,  the  right  de 

l-l 

e  canonical  derivation. 


ral,  a  parse  of  a  string  is  some  indicati 
was  derived.  In  particular,  a  canonical 
the  reverse  of  the  sequence  of  prod 
their  corresponding  #-symbols)  used  in 
f  a.  The  action  of  determining  a  parse 
e  determination  constitutes  a  grammatical 
g  algorithm  is  referred  to  as  a  parser  . 


immediate 
here  exists 
.  -  >  a  =  a 
,  is  one  in 
vable  from 
rightmost 
rivation  is 

on  of  how 
_parse  of  a 
uction  (or 
a  canonical 
is  called 
_analysis  , 


An  alternative  definition  for  the  left  part  of  a  production 

is  a  production _ goal  (PG) .  The  left-most  symbol  in  the  right 

part  of  a  production  is  referred  to  as  the  prgduction_head  (PH) 
of  the  production  goal,  and  the  right-most  symbol  in  the  right 
part  of  a  production  is  referred  to  as  the  prgduction_ta il  (PT) 
of  the  production  goal.  The  #-symtol  is  not  considered  in  the 
definition  of  PH  or  PT.  A  produced  production  head  (FPH)  of  a 
production  goal  is  either  its  production  head,  or  a  production 
head  of  one  of  its  produced  production  heads.  A  produced 

production _ tail  (PPT)  of  a  production  goal  is  either  its 

production  tail,  or  a  production  tail  of  one  of  its  produced 


, 
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production  tails.  The  number  of  recursions  involved  in  the 
definition  of  a  PPH  (PPT)  of  a  production  goal  is  referred  to  as 

the  level _ of  production  .  A  PH  (PT)  of  a  production  goal  has  a 

level  of  one. 

A  finite  state  machine  (FSM) ,  informally,  is  an  entity 
composed  of  states  and  transitions  out  of  these  states.  A 
transition  consists  of  a  transition _ symbol  (a  terminal,  non¬ 

terminal,  or  a  #-symbol) ,  and  a  destination  state.  The 
transition  symbol  is  so  named  because  in  a  given  state,  passing 
over  the  symbol  (reading  it  or  applying  a  production)  defines 
the  state  transition.  For  the  purposes  of  the  parser  generator 
itself,  four  different  transitions  are  possible:  read 
transitions  corresponding  to  the  reading  of  a  terminal  or  a  non¬ 
terminal,  #- transit ions  corresponding  to  applying  a  production, 

lookahead _ transitions  corresponding  to  looking  ahead  at 

terminals  and  push _ transitions  corresponding  to  generating  a 

non-terminal  without  having  considered  any  input  symbol. 

A  read  transition  of  a  terminal  symbol  is  called  a  terminal 
transition  ,  and  a  read  transition  of  a  non-terminal  symbol  is 
called  a  non- ter jninal_transi tion^ 

Any  state  having  read  transitions  is  called  a  read _ state.. 

Any  state  whose  only  transition  is  a  #-transition  is  called  a 
reduce_state  (or  an  a£ply_prod uc tion_sta ta  ) .  Any  state  with  a 
push  transition  is  a  push_state  and  any  state  with  lookahead 
transitions  is  a  lookahead_state  . 
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Any  read  state  containing  two  or  more  transitions  at  least 
one  of  which  is  a  #-transition  is  called  an  inadequate  state  . 
It  is  inadequate  in  the  sense  that  applying  a  production  is 
inconsistent  with  applying  other  productions  or  reading  symbols 
at  the  same  time. 

State _ generation  consists  of  specifying,  for  each 

transition  in  the  state,  the  transition  symbol  and  the 
destination  state.  Specifying  all  transitions  for  all  the  states 

is  called  machine _ generation  .  A  machine  without  inadequate 

states  is  an  LBJO} _ machine  . 

Every  read  state  or  produce  state  is  identified  uniquely  by 
a  configuration.  A  configuration  is  a  set  of  marked  productions 
each  of  which  identifies  a  transition  symbol.  The  production  is 
®§£ked  in  the  sense  that  one  and  only  one  symbol  on  the  RHS  of 
the  production  inclusive  of  the  #-symbol  is  tagged.  This  tagged 
symbol  is  a  transition  symbol  for  the  state  identified  by  this 
configuration. 

The  act  of  comgleting  a  configuration  consists  of  adding 
new  marked  productions  to  the  existing  set  of  marked  productions 
in  an  iterative  or  recursive  manner.  For  each  tagged  non¬ 
terminal  symbol,  the  set  of  productions  with  this  non-terminal 
symbol  as  production  goal  is  added,  with  the  production  head 
marked. 

As  an  example  of  completing  a  configuration,  consider  the 


following  grammar: 


- 

. 


A  ->  BC#  (1) 
X  ->  Y#{2) 

B  ->  A#  (3) 
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B  ->  X #  (4) 

and  the  following  configuration: 

(A  ->  B  C  #(1),  X  ->  Y  £12i  }. 

The  tagged  symbols  are  underlined.  Since  B  is  a  non¬ 
terminal,  two  new  productions  (3  and  4)  are  added  with  the 
production  head  tagged.  The  configuration  then  consists  of 
{  A  ->  B  C  #  ( 1 )  ,  X  -  >  Y  #  J2}_  ,  B  ->  A  #  (3)  , 

B  ->  X  #  (4)  }. 

Since  A  is  a  non- terminal,  the  marked  production 
A  ->  B  C  #  (1)  should  be  added,  but  it  already  exists.  The 
tagged  X  implies  X  ->  Y  #  (2)  is  to  be  added.  The  completed 
configuration  is  therefore: 

{A  ->  B  C  #(1),  B  ->  A  #  (3 )  ,  B  ->  X  #  (4)  , 

X  ->  Y  #  (2)  ,  X  ->  Y  #£2].  }. 

The  configuration  before  completion  is  called  the  initial 
configuration  .  A  set  of  configurations  is  called  a 
£2Bf igu ra t ion_se t  . 


• 
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2_-_BACKGR0UND_M ATE RIAL 


2 . 1  The  DeRemer  Approach 


The  DeRemer  approach 
state  stack  (to  keep  t 
state  parser.  This  finite 
corresponding  directly  to 
The  LALR  (k)  machine  for  k 
extension  of  the  LR  (0)  mac 


basi 

ca 

iiy 

inc 

ludes 

the  mer 

ging 

he 

pa 

rse 

his 

tory) 

along  wi 

th  a 

state 

parser 

can 

be  run 

off 

the 

L 

ALR 

(k) 

machi 

ne  for  th 

e  la 

>=  1 

can 

be 

sim 

ply  deri 

ved 

hine  for  that  language. 


of  a 
finite 
tables 
nguage. 
as  an 


The  generation  of 
LALR  (k)  is  subject  for 
an  example  of  an  LR  (0) 
LALR  ( 1)  machine  for  an 


the  LR  (0)  machine  and  the  extension  to 
later  discussion.  At  this  point,  however, 
machine  followed  by  its  extension  to  an 
L R  ( 1 )  language  might  be  more  appropriate. 
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An_L  R_(Oj _ Machine 
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I7l - -H3 


** 
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<G0 AL>  ->  E  _ |_  #  (1) 
E  ->  E  +  T  #  (2) 

E  ->  T  #  (3) 

T  ->  P  **  T  #  (4) 


T  ->  P  I  (5) 

P  ->  I  ♦  (6) 

P  ->  (E)  #  (7) 
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An  Extension  From  LR  (0) _ To  LALR(1) 


Two  new  states  have  been  added,  namely  14 
together  duplicate  the  activity  of  the  old  state 


and  1 5 
7.  State 


which 
7  •  has 


been  made  into  a  lookahead  state 
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2.2  The  Knuth-Earley  Algorithm 


This 
example 
given  in 
steps: 

1) 


is  bas 

ically 

a 

very 

si  IB 

pie 

algori 

thm. 

A  n 

informal 

of  the 

algor 

ithm 

as  a 

PPli 

ed  t 

o  gener 

ating 

a  m 

achine  is 

the  app 

endix. 

For 

mally 

,  it 

con 

sists 

of  the 

following 

Collec 

t  all 

prog 

ram  g 

oal 

prod 

uct ions 

and 

mark 

each 

produc 

tion 

hea 

d. 

This 

f 

orms 

the 

ini 

tial 

conf ig 

uratio 

n. 

Compl 

ete 

the 

con  f  i 

gurat 

ion 

and 

place 

it 

in 

a 

con 

f  igu 

ration 

set 

as 

the 

conf ig 

uratio 

n  u 

nigue 

iy 

iden 

tif  yi ng 

ung 

enerated 

state  1. 


2)  Search  the  configuration  set  for  a  configuration 
corresponding  to  an  ungenerated  state.  If  none 
exists,  the  machine  is  generated.  Otherwise, 
carry  on. 

3)  Is  the  state  completely  generated  (is  there  an 
identical  transition  symbol  out  of  the  state  for 
each  marked  symbol  in  the  configuration)  ?  If  the 
state  is  fully  generated,  go  to  #2.  Otherwise, 
carry  on. 

4)  Choose  any  marked  symbol  which  is  not  yet  a  transition 
symbol  from  that  state.  If  the  marked  symbol  is  a 
♦-symbol,  go  to  #5;  otherwise,  go  to  #6. 


' 


‘ 
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5)  A  #-transition  is  created  with  the  #-symbol  as  the 
transition  symbol  and  with  a  unique  "restart” 


st 

ate 

as  the 

destination  state  (an 

FSM 

parser  has 

no 

stack  and 

hence  the  modified  input 

string  must 

be 

re- 

parsed 

after  each  production 

is 

applied) . 

Go 

to 

#3. 

6)  A  read  transition  is  created  with  the  marked  symbol  as 
the  transition  symbol.  The  destination  state  must 
be  determined  by  forming  the  "destination” 
configuration.  Its  initial  configuration  is 

created  by  taking  the  union  of  all  productions 
(in  the  original  configuration  for  this  partially 
generated  state)  in  which  the  transition  symbol 
is  the  marked  symbol,  and  moving  the  mark  to  the 
right  one  symbol.  The  configuration  is  then 
completed.  If  it  already  exists  in  the 

configuration  set,  the  destination  state  is 
known.  Otherwise,  the  configuration  is  inserted 
in  the  configuration  set  as  uniquely  identifying 
a  new  ungenerated  state  (which  is  given  an  unused 
state  number).  Again,  the  destination  state  is 


known.  Go  to 


#3. 


* 


■ 

■ 
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3_-_Global_Design_of _the_Ef f icient_Par ser_Genpr at or 


3.1  General 


In  general,  the  LALR(k)  Parser  Generator  accepts  as  input 
an  LALR(k)  context  free  grammar  expressed  in  BNF ,  and  produces  a 
set  of  parsing  tables  usable  by  an  LALR  (k)  finite  state  parser. 


The  Parser  Generator  can  be  roughly  subdivided  into  6 
parts: 


1) 

Input  of  the  Source 

Grammar, 

2) 

Initial  grammatical 

analysis , 

3) 

Generation  of  an  LR 

(0)  machine. 

4) 

Extension  to  SLR(1) 

or  LALR ( k)  through  local 

lookahead , 

5)  Optimization  of  the  LALR  (k)  machine  and  its  conversion 
to  a  DPDA, 

6)  Transformation  of  the  DPDA  to  tables  (with  more 
optimization) . 


3.2.1  Sour ce_Gr a mmar 


In  order  to  have  the 
length  with  symbols  of 
arbitrary  characters,  the 


ability  to  input  productions  of  any 
reasonable  length  and  of  relatively 
free  format  style  of  Van  Wijngaarden 


[5]  was  chosen.  The  result  was  that 


. 


' 
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1) 

2) 

3) 

<0 


5) 


all  blanks  are  ignored, 

all  tokens  are  delimited  by  or  , 

a  token  consists  of  from  1  to  62  non-blank  characters 
without  an  imbedded  delimiter, 

a)  :  implies  the  preceding  was  a  production  goal, 

b)  ;  implies  the  start  of  a  new  production  with  the 

same  production  goal  as  the  previous 
production  (preceding  token  is  the  tail  of 
preceding  production) , 

c)  ,  implies  preceding  token  is  followed  by  another 

token , 

d)  .  Implies  preceding  token  is  the  tail  of  preceding 

production, 

the  empty  symbol  is  denoted  by  ”<EMPTY>". 


E .  g .  1 

A  ->  BCD 
A  ->  E 
C  ->  G 
G  ->  F 

G  ->  <EMPTY> 
can  be  written  as 

A: B,C,D;E.C:G.G: F;<E«PTY>. 


. 

i  ' 
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e.g .  2 


LIST  ->  LABEL  VAR-  ,  VAR-  . 

LABEL  ->  ID*  : 
can  be  written  as 

LABEL:ID*,:.LIST:LABEL,VAR-,  ,  , VAR-,.. 
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3.2.2  Initial  Grammatical  Analysis 


Standard  techniques  exist  for  the  detection  of  simultaneous 
left  and  right  recursion  including  the  special  case  of  self¬ 
production  or  circularity. 


Simultaneous  left  and  right  recursion  results  in 
ambiguities  in  the  parse  since  more  than  one  distinct  parse  is 
possible  for  some  strings. 


Before  elaborating  on  an  algorithm,  consider  the  informal 
analysis  given  to  the  example  below. 


Given 


A  ->  BCD  #  ( 1) 


B  ->  AX  #  (2) 
C  ->  Y  #  (3) 

Y  ->  C  #  (4) 

D  ->  XE  #  (5) 


E  ->  Y  A  #  (6) 


then  B,  A,  Y,  C,  X,  Y  are  production  heads  of  each 
successive  production  and  D,  X,  Y,  C,  E,  A  are  production  tails 
of  each  successive  production. 

Also,  since  B  is  a  production  head  of  A  in  production  1, 
and  A  is  a  production  head  of  B  in  production  2,  then  A  in 
production  2  is  a  produced  production  head  of  A  in  production  1. 
Therefore  respective  produced  production  heads  (in  alphabetic 
order),  of  each  production  goal  are  (A,B),(A,B),  (C ,  Y)  ,  (C,Y), 

(X)  ,  and  (Y)  .  Similarly,  produced  production  tails  of  the 


* 
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respective  production  goals  are  (A,D,E),  (X)  ,  (C,Y),  (C,Y), 

(A,Df  E)  ,  and  (A,  D,  E)  . 

Since  the  produced  heads  and  produced  tails  of  production  1 
are  (A,B)  and  (A  ,  D ,  E)  respectively,  then  it  can  have  the  derived 
fora  A  ->  A _ A  and  is  hence  ambiguous. 

The  same  holds  for  production  3  which  has  a  Y  in  both  the 
produced  heads  and  produced  tails,  implying  in  this  case 
circularity. 

If  all  produced  heads  and  produced  tails  for  each  unique 
production  goal  are  collected,  the  occurence  of  left  and  right 

recursion_or_circulari ty_is_conf ir raed_when_t he _ production _ goal 

occurs _ in _ both _ its _ produced _ production _ heads _ and  produced 

prod act ion_t ails  . 

Moreover,  the  produced  production  heads  (tails)  of  all 
production  goals  can  be  calculated  very  easily  from  their 
initial  production  heads  (tails)  . 

Consider  a  bit  matrix  (LEFT)  of  non-terminals  (NT)  versus 
non-terminals.  Initialize  each  element  LEFT(i,j)  to  M'B  (on)  if 
the  j  th  non-terminal  is  a  production  head  of  the  i  th  non¬ 
terminal  and  *  0  *  B  (off)  otherwise.  Then  LEFT  is  a  matrix  of 
production  goals  versus  production  heads. 

If  we  iterate  LEFT  =  LEFT  *  LEFT  (where  *  is  the  matrix 
product)  until  it  stabilizes  (it  must  since  no  on  bit  is  ever 

matrix  of 


turned  off  in  an  iteration),  then  LEFT  is  now  a 
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production  goals  versus  produced  production  heads.  The  following 
argument  may  clarify  the  reasoning. 

Consider  the  definition  of  bit  matrix  product: 

LEFT  (i,  j)  =  \/  (LEFT  (i,k)  A  LEFT  (k,  j)  )  . 

VK 

This  implies  that  if  NT  (k)  is  a  PH  of  production  goal  NT  (i) 
and  NT { j)  is  a  PH  of  production  goal  NT (k)  ,  then  NT (j)  is  a  PH 
of  production  goal  NT  (i) .  Moreover,  with  one  iteration,  produced 
heads  to  the  second  level  are  found;  with  two  iterations,  each 
of  these  can  be  produced  two  more  levels  (since  the  level  of 
production  for  each  NT  is  currently  known  to  two  levels) ,  and  so 
on.  In  general,  only  k  iterations  are  required  to  find  all 
produced  heads  where  2**k  <=  m  (and  m  is  the  max  level  of 
production)  . 

In  a  similar  fashion  RIGHT  ,  a  matrix  of  production  goals 
vs  produced  production  tails  can  be  formed.  Then  the  scalar 
product  of  the  two  main  diagonals  of  these  matrices,  if  not 
zero,  implies  that  for  some  production  goal,  the  goal  symbol  is 
both  its  own  PPH  and  PPT,  indicating  left  and  right  recursion 
(  (^i)  [  LEFT  (i,i)  =  RIGHT  (i,  i)  ])  . 

The  addition  of  empty  right  parts  to  the  grammar  clearly 
requires  more  sophistication.  For  instance,  if  B  has  an  empty 
right  part,  then  its  use  in  a  production  like  X  ->  BYZ  indicates 
that  Y  can  also  be  a  production  head  (in  addition  to  the  B)  . 
Clearly  if  the  Y  has  an  empty  right  part,  the  Z  also  is  a 
production  head,  and  so  on. 


. 


- 
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A  list  of  production  goals  with  empty  right  parts  and 
produced  empty  right  parts  must  therefore  be  tabulated  (a 
relatively  slow  operation)  from  the  grammar.  No  easy  bit 
manipulation  methods  exist  since  each  production  must  be  scanned 
element  for  element  as  opposed  to  first  and  last  elements  only. 

Using  this  information,  the  modified  production  head  (tail) 
matrix  can  be  generated  and  the  standard  algorithm  finally  used 
to  obtain  the  produced  production  head  (tail)  matrix. 

It  is  desirable  to  check  the  grammar  at  this  stage  for 
multiple  program  goals  (only  one  goal  is  allowed). 
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3.2.3  Genera  tion_of_dn_LH_(0] _ Machine 

The  LB (0)  machine  of  DeRemer  can  be  derived  by  using  the 
Knuth-Earley  algorithm. 

In  order  to  implement  the  algorithm  easily,  however,  a 
simple  method  of  storing  configuration  sets  must  be  used.  One 
such  method  is  representation  by  means  of  a  bit  vector.  Each  bit 
by  its  positional  value  must  represent  an  element  on  the  RHS  of 
a  production  (where  the  RHS  consists  of  terminals,  non-terminals 
and  the  final  #-syrabol) .  All  productions  must  be  fixed  in  a 
specified  order  once  the  generation  of  the  LR (0)  machine  is 
begun.  The  conclusion  is  that  each  configuration  must  consist  of 
C#  bits  where 

Cf  =  number  of  terminals  and  non- terminals  on  RHS  of  all 
productions  ♦  number  of  productions). 

A  configuration  set  is  therefore  represented  by  a  list  of 
bit  vectors  or  equivalently  a  bit  matrix.  Each  state  is  then 
identified  by  its  configuration  and  fully  specified  when  it  has 
been  generated,  that  is,  filled  in  with  transition  information 
such  as  "read  A,  go  to  10”,  "read  B,  go  to  21",  "apply  5", 
"apply  7".  As  can  be  seen  from  the  informal  example  in  the 
appendix  (which  is  worthwhile  following  to  make  clear  the 
details  of  the  algorithm) ,  no  apparent  order  exists  during  the 
generation  of  the  LR  (0)  machine.  However,  a  simple  set  of  rules 
may  be  followed  to  give  it  an  algorithmic  flavour. 


. 
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1)  Set  S#  (state  number)  pointing  to  the  first  and  only 
initial  configuration  (the  goal  configuration)  in 
the  configuration  set  and  MAXS#  pointing  at  the 
last  ungenerated  configuration  (initially  the 
first) . 

2)  As  state  St  is  being  generated,  the  initial 
configuration  of  the  destination  or  "go  to”  state 
of  each  transition  item  in  turn  is  developed  and 
then  completed. 

3)  Each  of  these  completed  configurations  is  successively 
compared  with  existing  configurations  in  the 
configuration  set.  If  it  previously  exists,  then 

the  destination  state  is  known;  otherwise,  the 
ungenerated  configuration  is  added  to  the  list 
and  MAXS#  is  increased  by  one. 

4)  S#  is  increased  by  one  and  the  next  state  is 
generated • 

5)  The  machine  is  completed  (all  states  are  generated) 
when  S#  >  KAXSf. 

This  algorithm,  however,  will  later  complicate  the 
situation  when  read  states  and  reduce  states  must  be  sorted  (for 
efficiency,  to  be  subsequently  remarked  on).  The  solution  is  to 
simply  initialize  the  configuration  set  with  more  than  the 
program  goal  configuration;  namely,  the  configurations  for  all 
the  reduce  states  followed  by  the  program  goal  configuration). 
This  effectively  forces  the  states  from  1  to  P#  (production 
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number)  to  be  reduce  states  for  (apply  1,  apply  2,  . ..,  apply 

P#)  . 

Another  convenience  is  to  add  to  each  state  (except  the 
start  state)  the  name  of  the  terminal  or  non-terminal  which  was 
read  in  order  to  access  it. 

If  the  machine  is  completely  generated  without  any 
inadequate  states,  then  the  input  language  is  LR  (0) .  Recall  that 
inadequacy  here  refers  to  the  fact  that  multiple  #- transitions, 
or  #- transitions  and  read-transit ions  cannot  occur 
simultaneously  (hence  the  requirement  for  a  state  just  ahead  of 
the  inadequate  state  to  resolve  the  ambiguity) . 

On  the  other  hand,  if  inadequate  states  exist,  look-ahead 
and  possibly  push  states  (for  empty  right  parts)  are  necessary 
to  resolve  the  ambiguity  inherent  in  the  inadequate  state. 

Empty  right  parts  will  automatically  generate  inadequate 
states  since  the  production  goal  and  the  #-symbol  for  that 
production  will  occur  as  transitions  in  the  same  state. 


. 
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3.2.4.  1  Ex  te  nsion_to_S  LR  J2)_ 


Extension  ot  an  LR  (0)  machine  with  inadequate  states  to 
SLR  (1)  is  done  very  simply.  A  certain  amount  of  information, 
however,  must  first  be  gathered.  Fortunately,  this  information 
exists  in  the  LR  (0)  machine. 


For  each  non-term 


which 

can 

follow 

it 

by  running 

through 

eac 

symbol 

was 

read  in 

ord 

inal,  a  list  of  terminals  and 
is  tabulated.  This  information 
h  state  in  the  LR  (0)  machine  kn 
er  to  access  that  state. 


♦-symbols 
is  gathered 
owing  which 


Each  f-symbol  is  then  deleted  and  replaced  by  the  symbols 
(terminals  and  #-symbols)  which  can  follow  the  production  goal 
corresponding  to  that  #-symbol.  This  is  obviously  very  recursive 
but  fortunately,  it  can  be  done  iteratively  with  the  same  bit 
matrix  multiplication  technique  as  used  in  detecting  obvious 
ambiguities. 


For  example,  we  might  find  that  following  non-terminal  A  we 
can  have  terminals  X,  ♦ ,  *  and  Y  and  #-symbols  # (2)  and  # (5) . 


If 

the  #-symbols  refer 

to 

B  ->  ZZ#  (2) 

and  C  ->  W# (3) 

then 

we 

know  that  following 

A  we  can  have  the 

ter mi nals 

Y  4.  * 

and 

Y 

and  the  terminals 

that  can  follow 

either  of 

the  non- 

terminals  B  and  C 
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When  the  full  list  is  compiled ,  we  can  run  through  the 
machine  looking  for  inadequate  states  and  resolving  each  one 
individually  by  means  of  a  lookahead  state  which  must  precede 
the  inadequate  state.  The  inadequate  read  state  in  turn  is 
transformed  into  an  "adequate”  read  state  and  one  or  more  reduce 
states. 

For  each  terminal  transition  in  the  inadequate  state,  we 
add  the  same  to  the  lookahead  state.  The  destination  of  these 
transitions  is  the  inadequate  state  which  will  later  be  trimmed 
to  a  read  state. 


For  each  #-transi tion ,  we  add  the  list  of  terminals  which 
can  follow  the  production  goal  corresponding  to  that  #-symbol. 
The  destination  state  for  a  # (P)  symbol  is  apply  production 
state  P  (unless  production  P  has  an  empty  right  part  in  which 
case  the  destination  state  is  a  push  state  which  pushes  the  name 
of  the  inadequate  state  in  the  state  stack  and  then  makes  an 
implicit  non-terminal  transition  to  apply  production  state  P) . 
The  reason  for  the  push  state  will  be  further  elaborated  upon  in 
section  3.2.5. 


If  at  any  time,  identical  transitions  with  different 
destinations  are  entered  in  the  lookahead  state,  then  at  least 
two  transitions  in  the  inadequate  state  cannot  be  distinguished, 
and  therefore  the  inadequate  state  can  not  be  resolved  via  the 
SLR  ( 1)  algorithm. 


. 


1 


■ 


26 


If  all  states  can  be  resolved,  then  the  grammar  is  SLR(1); 
otherwise,  it  is  not. 

Notice  that  this  is  a  global  technique  based  on  what  can 
follow  a  particular  read  item  in  general  as  opposed  to  the 
particular  local  environment  of  that  inadequate  state.  This 
deficiency  is  removed  in  the  LALR(k)  algorithm. 

Although  the  SLR(1)  algorithm  is  implemented,  the  more 
general  SLR  (k)  algorithm  is  not,  simply  because  the  complexity 
of  the  algorithm  approached  in  part  that  of  the  LALR  (k) 
algorithm  which  in  turn  supercedes  the  SLR  (k)  technique. 
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3.2.4. 2  te nsion_to_L AL R_(k}_ 

The  LALR  (k)  algorithm  has  two  advantages  over  the  SLR(1) 
algorithm,  namely 

1)  it  offers  the  ability  to  look  ahead  for  k  symbols 
(k  >=  1)  . 

2)  it  is  based  on  local  look-ahead. 

The  philosophy  of  the  LALR  routine  is  such  that  when  a 
state  is  inadequate,  a  tree  of  predecessor  states  which  can 
access  this  inadequate  state  is  built  up  one  level  at  a  time 
(one  level  meaning  one  transition) .  To  each  level  there 
corresponds  a  predecessor  set.  The  depth  or  number  of  successive 
predecessor  sets  which  must  be  calculated  depends  naturally  on 
the  maximum  number  of  states  which  must  be  pulled  (by  applying  a 
production)  starting  originally  from  the  inadequate  state.  These 
predecessor  sets  can  therefore  be  considered  as  forming  a 
mainline  predecessor  path  (in  actual  fact,  a  predecessor  tree) 
which  dictates  the  past  history  up  to  this  particular  inadequate 
state.  From  any  given  state,  if  no  #-symbols  are  encountered,  it 
is  an  easy  matter  to  project  forward  to  obtain  sequential  lists 
of  k  terminals  which  could  be  seen  by  lookahead.  When  a  #(P)- 
symbol  is  encountered,  however,  the  terminals  which  could  follow 
if  production  P  were  applied  must  be  collected.  To  do  this,  the 
number  of  states  equal  to  the  length  of  the  RHS  of  production  P 
is  pulled.  Furthermore,  of  the  possible  states  which  are  now 
visible,  only  those  which  can  reach  the  inadequate  state  (name  ly 
those  on  the  mainline  predecessor  path)  are  candidates.  Having 
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found  the  production  goal  for  production  P  in  each  of  these 
states,  the  process  of  collecting  terminals  is  resumed  starting 
from  each  of  the  destination  states  of  these  production  goals. 
These  terminals  are  of  course  added  on  to  the  successive 
terminals  collected  so  far.  This  is  obviously  very  recursive  and 
repetitive  but  nevertheless,  essential  for  localized 
inadequacies. 

Moreover,  if  during  look-aheads,  side  branches  are  followed 
which  lead  away  from  the  mainline  predecessor  path  (this  occurs 
whenever  a  symbol  is  added  to  the  set  and  at  least  one  other 
succeeding  symbol  is  sought) ,  then  pulls  which  are  performed 
there  must  fall  on  the  side  branch  taken  or  (if  the  pull  has 
enough  depth)  on  the  mainline  predecessor  path. 

In  all  cases,  the  mainline  predecessor  path  is  backed  up 
dynamically  as  far  as  necessary. 

At  this  point,  it  might  be  wise  to  consider  an  example 
which  is  L ALR  ( 1)  but  not  SLR(1).  The  following  grammar  is  one 
such  example: 

<GOAL>  ->  S  _J_  #  ( 1) 

S  -  >  E  =  E  #  (2) 

S  ->  P  #  (3) 

E  ->  T  #  (4) 

E  ->  E  +  T  #  (5) 

T  ->  P  #  (6) 

T  ->  T  *  P  f  (7) 


*• 


. 
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The  corresponding  machine  without  writin 
configuration  for  each  state  is  as  follows, 
trouble  spot  in  this  grammar  lies  in  the  fact  that 
can  be  a  P,  the  E  also  can  be  a  P. 

Read  State  1  Accessed  With  ***** 

P  go  to  5 
E  go  to  6 
S  go  to  8 
T  go  to  9 

Read  State  2  Accessed  With  + 

P  go  to  16 
T  go  to  10 

Read  State  3  Accessed  With  * 

P  go  to  17 

Read  State  4  Accessed  With  = 

P  go  to  16 
E  go  to  7 
T  go  to  9 


Read  State  5  Accessed  With  P 


g  out  the 
The  essential 
although  S 


#  (3)  Pull  1 

#  (6)  Pull  1 

Read  State  6  Accessed  With  E 

♦  go  to  2 
=  go  to  4 

Read  State  7  Accessed  With  E 

♦  go  to  2 

f  (2)  Pull  3 
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Bead  State  8  Accessed  With  S 
_  j  _  go  to  11 

Read  State  9  Accessed  With  T 

*  go  to  3 

#  (4)  Pull  1 

Bead  State  10  Accessed  With  T 


*  go  to 

i  3 

#  (5)  Pull  3 

Apply  Production 

State  11 

Accessed 

With 

# (1)  Pull  2 

(produce 

<GOAL>) 

Apply  Production 

State  12 

Accessed 

With 

£ 

#  (2)  Pull  3 

(produce 

S) 

Apply  Production 

State  13 

Accessed 

With 

P 

# (3)  Pull  1 

(produce 

S) 

Apply  Production 

State  14 

Accessed 

With 

T 

#  ( 4)  Pull  1 

(produce 

E) 

Apply  Production 

State  15 

Accessed 

With 

T 

# (5)  Pull  3 

(produce 

E) 

Apply  Production 

State  16 

Accessed 

With 

P 

#  (6)  Pull  1 

(produce 

T) 

Apply  Production 

State  17 

Accessed 

With 

P 

# (7)  Pull  3 

(produce 

T) 

The  problem  manifests  itself  in  the  first  inadequate  state 
(state  5) . 

The  SLR  approach  attempts  to  resolve  between  the  #  (3) 
transition  and  the  #  (6)  transition  by  the  introduction  of 
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lookahead.  A  table  of  what  can  follow  each  non-terminal  except 
the  goal  is  created.  The  results  are: 

S  followed  by  {_!_} 

E  followed  by 

T  followed  by 

Consider  first  the  #  (3)  transition.  If  production  3  is 
applied,  an  S  is  produced  and  it  can  be  followed  only  by 

Considering  next  the  #  ( 6)  transition,  if  production  6  is 
applied,  a  T  is  produced  which  can  be  followed  by  {*,*,  =  _  }• 

Clearly  the  ”_J_”  occurs  in  both  and  hence  the  # (3)  transition 
cannot  be  distinguished  from  the  #  (6)  transition. 

□sing  the  LALR  approach  however,  the  situation  is  more 
fortunate.  For  the  #  (3)  transition,  the  same  lookahead  set 
is  obtained.  Consider  therefore  the  #(6)  transition  in  state  5. 
Initialize  the  zeroth  predecessor  set  with  {5}  and  set  a  pointer 
to  this  predecessor  set  remembering  that  the  current  state  is 
state  5,  and  that  no  symbols  have  been  "looked  at”.  When 
production  6  is  applied,  one  state  is  pulled  from  the  state 
stack.  The  states  that  are  predecessors  to  the  zeroth 
predecessor  set  are  therefore  the  set  [1}  (those  reading  P  and 
going  to  state  5  since  state  5  is  accessed  by  reading  P) .  The 
pointer  is  backed  up  one  and  points  at  this  last  predecessor 
set.  The  application  of  production  6,  however,  produces  a  T. 
Advancing  across  the  T  in  state  1  to  state  9  cause  the  pointer 
to  be  incremented  by  one  and  to  point  to  the  original 
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predecessor  set.  In  state  9,  the  n*n  is  picked  up  and  placed  in 
the  lookahead  set  which  now  contains  {*}.  The  next  item,  the 
f  (4)  symbol  requires  a  pull  of  1 ,  decreasing  the  pointer  by  1 
(the  predecessor  set  has  of  course  been  previously  calculated 
and  consists  of  { 1  }) .  Application  of  production  4  produces  an  E. 
On  moving  across  the  E  to  state  6,  the  pointer  is  incremented  by 

1  and  points  at  the  original  zeroth  predecessor  set.  In  state  6, 

2  items  are  added  to  the  lookahead  set  yielding  {+,*,=}.  Nothing 
else  exists  and  the  inadequacy  is  resolved. 

If  the  lookahead  symbols  are  {_!_},  a  # (3)  transition  must 
follow  and  if  the  lookahead  symbols  are  {•*-,*,=  },  a  #  (6) 
transition  must  follow. 

The  reason  why  the  LALR  technique  was  able  to  resolve  state 
b  is  fairly  simple.  From  state  9,  the  application  of  production 
4  implied  that  the  candidate  predecessor  states  consisted  of 
member  of  the  set  {  1,3}.  If  the  path  from  state  3  were  pursued, 
the  symbol  would  be  a  possible  lookahead  item.  However, 
state  3  could  not  be  a  predecessor  since  the  predecessor  set  at 
this  depth  (calculated  using  the  previous  predecessor  set  i.e. 
The  zeroth  in  this  case)  consisted  only  of  {1}  (the  only  state 
which  could  reach  state  5) . 
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3.2.  5 


Machine  Optimization  and  Transformation  to  a  DPDA 

Two  related  major  optimizations  can  be  done  at  this  point: 

1)  creation  of  a  more  powerful  reduce  state  which  would 
allow  a  parse  to  be  continued  after  an  apply 
production, 

2)  deletion  of  all  non- terminals  in  the  machine. 

A  more  powerful  reduce  state  insures  with  the  aid  of  a 
state  stack  (speaking  in  terms  of  the  parser)  that  popping  a 
fixed  number  of  states  for  each  production  allows  us  to 
determine  (from  the  information  on  the  state  stack)  which  state 
to  resume  the  parse  in.  This  in  effect  converts  the  LALR  (k)  FSM 
to  an  LALB  (k)  DPDA. 

If  all  non- terminals  were  kept,  a  reduce  state  would  merely 
have  to  carry  the  number  of  states  to  pull  from  the  state  stack. 
Having  applied  the  production  and  produced  a  non-terminal,  this 
non-terminal  could  be  re-read  by  the  parser  to  carry  on. 

It  is  however  clear  that  re-reading  this  non-terminal  could 
be  bypassed  if  a  correspondence  table  of  some  sort  existed. 
Since  the  number  of  states  to  pull  from  the  apply  production 
state  is  known,  a  list  of  all  those  states  which  could  now  be  on 
top  of  the  state  stack  can  be  compiled;  say  states  10,  22,  99 

are  eligible.  Since  these  states  read  this  non-terminal 
(production  goal) ,  then  the  destination  state  is  known.  Consider 
them  as  being  4,  21,  52  respectively.  The  reduce  state  could 


then  appear  as  follows: 
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Apply  Production  State  i 

Apply  p,  Pull  g 

With  the  following  on  the  state  stack, 

go  to  the  indicated  state: 

10  go  to  4 
22  go  to  21 
99  go  to  52 

At  this  point,  however,  a  certain  inefficiency  arises.  This 
can  be  shown  in  the  following  three  productions. 

A  ->  B  #  (1) 

B  ->  C  #  (2) 

C  ->  D  #  (3) 

Consider  that  MD”  has  just  been  read.  The  #  (3)  symbol 
should  then  be  pushed  into  the  parse  stack  and  immediately 

followed  by  a  pull  1  and  apply  3.  Having  produced  a  "C",  a 

transition  is  made  to  a  state  which  pushes  1  for  the  # ( 2)  symbol 

and  immediately  pulls  1  and  applies  2  followed  by  a  transition 

to  a  state  which  pushes  1  for  the  #(1)  symbol  and  immediately 
pulls  1  and  applies  1. 

This  needless  pushing  and  pulling  can  be  very  neatly 
avoided  if  only  read  states  are  allowed  to  push  and  reduce 
states  (because  they  themselves  miss  being  pushed  into  the  state 
stack)  pull  one  less  than  the  normal  pull.  The  only  complication 
arises  for  the  case  of  empty  right  parts  which  pull  zero  to 
start.  Clearly,  a  reduce  state  for  a  production  with  empty  right 
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part  before  optimization  has  to  push  the  reduce  state  and  pull 
zero.  To  keep  the  push  and  pull  count  correct  after 
optimization,  we  need  merely  push  the  name  of  the  read  state  in 
which  the  #-transition  occurred  into  the  state  stack  and  then 
pull  0.  This  ’’pushed”  state  name  will  in  fact  refer  to  an 
inadequate  state  which  had  been  resolved  into  an  ’’adequate”  read 
state. 

Having  done  this,  all  non-terminals  can  be  deleted  from  the 
DPD A .  This  significantly  decreases  the  size  of  each  state  (often 
by  as  much  as  a  factor  of  2) . 

A  third  optimization,  useful  in  increasing  the  parsing 
speed  and  reducing  table  information  is  to  sort  all  the  states 
in  a  specified  order;  namely  read  states  followed  by  lookahead 
states  followed  by  push  states  followed  by  apply  production 
states.  This  implies  that  the  kind  of  state  the  parser  is 
dealing  with  can  be  determined  merely  by  knowing  the  boundary  of 
each  of  these  four  kinds  of  states. 

At  the  same  time,  for  the  user’s  convenience,  each  group  of 
states  except  the  apply  production  states  are  sorted  internally 
according  to  the  symbol  passed  in  accessing  that  state. 
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3.2.6  Transf ormations_f rom  DPDA  to  Tables 

In  order  to  represent  the  DPDA  by  a  compact  set  of  tables, 
a  system  of  indirect  addressing  can  be  used.  This  will  also 
insure  that  identical  or  top-most  subset  states  are  not 
duplicated.  Two  parallel  arrays  can  indicate  the  start  and 
length  respectively  of  each  state  stored  independently  in  other 
parallel  arrays.  The  information  arrays  will  contain 
respectively  read-symbols,  lookahead  symbols,  or  top  of  state 
stack  values,  and  their  corresponding  destination  state. 

In  order  to  keep  the  width  of  the  indexing  array  small,  8 
bit  as  opposed  to  16  bit  numbers  if  possible,  the  information 
related  to  the  three  types  of  states  mentioned  above  can  be 
partitioned  into  three  sets  of  corresponding  arrays.  Clearly  the 
indexes  will  be  smaller  than  if  they  were  all  merged  into  one 
big  array. 

Push  states  can  be  built-in  to  the  index  arrays  since  the 
only  information  relevant  is  a  "push”  state  item  and  a 
destination  state  item. 

At  this  point,  the  pull  number  for  the  apply  production 
states  must  be  kept  in  an  auxiliary  array.  Further  optimization 
as  described  in  the  section  Further^Optimi zations  will  however 
remove  this  auxiliary  array  by  keeping  the  information  in  the 


index  array. 
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4_- _F ur ther_Opt i mlza t ions 

One  very  effective  space  optimization  at  this  point  is  to 
note  that  all  apply  production  states  with  the  same  production 
goal  must  of  necessity  be  the  same  (except  for  the  pull  number) . 
Therefore,  only  one  copy  need  be  kept  in  the  table. 

Another  space  optimization  is  allowed  by  option.  This 
includes 

1)  looking  for  identical  states  (apply  production  states 
excepted) 

2)  looking  for  top-down  subsets. 

Two  or  more  states  in  the  optimized  machine  may  in  fact  be 
identical.  This  results  because  the  initial  differences  between 
the  two  states  resided  only  in  non-terminals  (which  are  now 
removed) .  These  two  or  more  identical  states  may  therefore 
occupy  the  same  table  space.  They  may  not  be  merged  into  the 
same  state  unless  the  symbol  passed  accessing  each  of  these 
states  is  the  same.  The  situation  occurs  so  infrequently, 
however,  that  a  merge  routine  was  not  considered  worthwhile. 

A  free  gift  in  looking  for  identical  states  is  the  ability 
to  find  read  states  which  are  top-most  subsets  of  other  read 
states;  that  is,  with  the  particular  fashion  in  which  each  state 
is  sorted  internally,  it  is  possible  to  find  states  which  can  be 
fitted  wholly  within  another  state  from  the  first  symbol  onward 
without  changing  the  sort.  This  is  obviously  a  very  specialized 
type  of  subset  but  nevertheless  easily  found.  A  strict  subset  on 


' 


' 

. 


Jr  i  04  i  wmt  i  1 


38 


the  other  hand  would  involve  internal  state  reshuffling  and 
would  be  slightly  slower. 

A  third  optimization  can  be  done  to  all  look  ahead  and 
apply  production  states.  The  optimizations  are  in  fact  done  as 
the  expanded  states  are  created. 

Consider  first  the  lookahead  states.  They  contain  a  list  of 
all  symbols  that  can  be  validly  seen  via  lookahead  at  that 
particular  point  in  the  parse.  It  is  therefore  possible  to 
detect  errors  in  the  lookahead  state.  However,  parsing  speed  can 
be  increased  and  lookahead  state  size  can  be  decreased  if  error 
detection  is  delayed  until  a  subsequent  read  state.  In  other 
words,  if  we  admit  an  ”In  Any  Case”  condition  for  the  most 
popular  destination  state  in  the  lookahead  state,  we  find  that 
the  lookahead  state  can  change  in  size  by  up  to  an  order  of 
magnitude.  This  is  because  most  lookaheads  differentiate  between 
two  paths,  one  of  which  must  be  taken  for  a  small  number  of 
lookahead  tokens  (usually  one  or  two)  and  the  other  which  must 
be  taken  for  all  the  remaining  tokens. 

For  example,  consider  a  language  in  which  a  subset  of  its 
grammar  for  simple  arithmetic  expressions  is  of  the  form 


V 


- 


’ 

- 


39 


<Term> 


->  <Factor> 


#  (10) 


<Term>  ->  <Term>  ♦  <Factor> 


#  (11) 


<Ter m> 


->  <Term>  -  <Factor> 


#(12) 


<Factor>  ->  <Priraary> 


#(13) 


<Factor>  ->  <Factor>  *  <Priraary>  #  (14) 


<Factor>  ->  <Factor>  /  <Primary>  #(15) 


Also,  assume  that  an  arithmetic  expression  with  only  one 
variable  (no  operators)  is  being  parsed,  such  that  the  partial 
parse  at  this  point  is  <Primary>.  There  is  no  choice  here  but  to 
change  it  to  <Factor>  and  no  look  ahead  is  involved.  However,  at 
this  point,  it  is  necessary  to  differentiate  between  reading 
or  "/"  and  applying  production  10.  This  requires  a  lookahead 
state . 

For  the  possible  two  branches,  the  "read  *  or  /"  branch 
would  be  taken  if  a  "*"  or  a  "/"  were  the  lookahead  tokens  but 
the  apply  10  branch  would  be  taken  if  the  lookahead  tokens  were 
any  of  I !,  =  ,>=,<  =  ,-•=,>,<,;  }  or  any  other  symbols  which 

could  follow  a  <Term>. 

The  only  apparent  difficulty  with  the  above  optimization  is 
that  delayed  error  detection  might  in  fact  allow  certain  very 
obscure  errors  to  pass  undetected.  This  in  fact  is  not  true.  If 
the  look  ahead  symbol  were  not  in  the  original  complete 
lookahead  set,  and  the  modified  "In  Any  Case"  lookahead  set  was 
being  used,  the  "In  Any  Case"  exit  would  be  taken.  However,  a 
subsequent  read  state  would  eventually  be  forced  to  read  the 
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next  token.  This  token  in  fact  would  not  be  legal  in  any 
subsequent  read  state  since  it  was  not  included  in  the  look 
ahead  state  (the  look  ahead  state  is  created  by  looking  at  the 
surrounding  read  states) .  Therefore,  the  error  will  still  have 
been  detected  and  all  is  well. 

In  a  similar  manner,  the  apply  production  states  can  be 
considerably  compacted  by  the  addition  of  an  ”In  Any  Case” 
condition  for  the  most  popular  destination  state.  This  results 
in  absolutely  no  change  in  the  behaviour  of  the  parser  (since 
apply  production  states  do  not  detect  errors)  but  again  has 
great  impact  on  space  and  time. 

Since  the  look  ahead  token  and  the  state  on  top  of  the 
state  stack  must  be  searched  for  in  independent  linear  lists, 
then  naturally,  the  parsing  speed  goes  up  somewhat  inversely 
proportional  to  the  size  of  these  lists.  Any  compaction  of  this 
sort  is  therefore  of  great  value. 
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5_-_LALRj[_1]__Parser_Al2or  it  hrn 

Having  at  this  point  generated  the  required  tables  to  parse 
a  given  grammar,  an  algorithm  must  be  generated  to  make  use  of 
them  in  an  actual  parsing  situation. 

In  order  to  satisfy  the  greatest  number  of  users,  the 
following  two  conditions  should  be  met: 

1)  sufficient  error  recovery  (detection  is  a  necessity) , 

2)  lookahead  only  when  necessary. 

The  first  condition  is  self-explanatory.  However,  the 
second  needs  more  clarification.  In  some  parsing  algorithms 
(e.g.  In  the  XPL  compiler) ,  the  next  token  is  kept  in  a  buffer 
where  it  can  be  "looked  at”  without  modification  or  where  it  can 
be  "read”  implying  that  it  is  destroyed  and  the  next  token  is 
placed  in  the  buffer.  Careful  consideration  implies  that  what  is 
really  being  done  is  "premature”  lookahead.  Consider  for  example 
what  happens  when  say  the  tail  token  of  the  program  goal  is 
encountered.  Having  read  this  token,  another  token  must  be  input 
to  satisfy  possible  lookaheads  which  clearly  will  not  be  needed. 
At  this  point,  for  instance,  XPL  returns  the  same  token  again. 
However,  the  situation  is  much  more  serious  for  "statement  at  a 
time"  parsers.  The  next  token  after  a  complete  statement  is 
actually  obliterated. 

This  inadequacy  can  be  very  simply  avoided  by  restricting 
the  input  of  the  "next"  token  to  situations  in  which  a  read  or  a 
lookahead  of  this  token  is  required.  The  loss  in  efficiency  is 
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not  serious. 

In  the  following  parsing  algorithm,  consider  all  variables 
as  mnemonically  self-explanatory. 

1)  Initialize  the  following  variables. 

PRESEN TESTATE  =  Start  State 

(usually  1  unless  a  lookahead  is  needed) 
PARSE_STACK_POINTER  =  -1  (Empty  Stack) 
NO_LOOK_AHEAD_DONE  =  TRDE 

2)  Find  out  the  kind  of  state  currently  in. 


Go 

to  (#  3 

read  state. 

#4 

apply  state. 

#5 

lookahead 

state,  #6  push 

state) 

depending  on 

PRES  ENT_ST AT  E. 

3 )  ta  te 

PARS E_ ST ACK_ POINTER  =  PARSE_STACK_POINTER ♦  1 
(make  room  for  token) 

Check  for  Stack  Overflow. 

ST  ATE__ST  ACK  (PARS  E_STACK_PO INTER)  =  P  RE  SEN  TESTATE  . 
IF  NO_LOOK_AH£AD_DONE  THEN  CALL  SCAN  (get  a 
token) 

Find  matching  token  in  READ1. 

If  found,  put  the  token  and  related  information 

in  the  parse  stack 

(location  PARSE_STACK_POINTER ) . 

Set  PRES ENT_S TAT E  to  corresponding  READ2  and  go 
to  #2.  , 

Otherwise,  give  an  error  message  and  call  the 


. 
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error  recovery  routine. 

Apply  State 

The  rightmost  item  in  the  stack  is  pointed  to  by 
PARSE_STACK_POINTER.  The  leftmost  item  for  this 
production  depends  on  the  production  length 
(stored  in  INDEX2) .  Therefore  set 

LEFTMOST_POINTER  and  R IG HT_ M OS T_ POINTER 

accordingly. 

Apply  production  (  PRESENT_STA TE  -  maximum  push 
state  #) ;  apply  production  states  are  sorted  last 
and  push  states  second  last.  Reset  the  stack 
pointer  to  the  production  head  which  is  replaced 
by  the  production  goal. 

PARSE_ST ACK_POINTER  =  LEFTHOST_POINTER . 

Compare  STATE_ STACK  ( PARSE_ST AC K_ POINTER)  with 
possible  top  of  stack  states  in  APPLY  1.  Having 
found  it  or  the  "In  Any  Case"  condition,  set  the 
present  state  to  the  corresponding  APPLY 2  and  go 
to  #2,  unless  the  corresponding  APPLY2  contains  a 
zero  which  is  the  code  to  stop  compiling  (the 
goal  has  been  reached) . 

Lookahead  State 

If  NO_LOOK_AHEAD_DONE,  then  we  must  call  the 
scanner  and  set  NO_LOOK_AHEAD_DONE  to  FALSE; 
Compare  the  token  with  elements  of  L00K1.  Having 
found  it  or  having  hit  the  "In  Any  Case" 
condition,  set  PRESEN T_STATE  to  the  corresponding 
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L00K2  item  and  go  to  #2. 

6)  Push  State 

Increase  PAR SE_STACK_ POINTER  by  one.  Check  for 
Stack  Overflow.  Push  a  state  in  the  state  stack, 
STATE__STACK  (P  ARS  E_ST  ACK_POINTER) 

=  INDEX2 (PRESENT_STATE) .  Set  the  next  state. 
PRESENT_STATE  =  INDEX  1  (PRESENT__STA  TE)  go  to  #2. 

The  corresponding  error  recovery  routine  must  first  give 
out  an  error  message.  That  is  to  say  that  it  must  be  able  to 
output  something  like 

Illegal  Symbol  Pairs:  ....  And  .... 

Partial  Parse  to  this  point  is  . 

Fortunately,  the  required  information  exists  in  the  state 
stack  and  in  the  current  token.  All  that  need  be  added  is  a 
correspondence  between  the  state  number  and  the  name  associated 
with  that  state  (the  symbol  read  in  order  to  access  it) . 

This  can  be  done  via  a  subscripted  variable  called 
mnemonically  STATE_NAME  which  contains  the  corresponding  alias 
of  the  state. 

Error  recovery  can  be  very  crudely  accomplished  via  two 
mechanisms: 

1)  flushing  of  subsequent  tokens  until  a  "hard"  token  is 
encountered  (certain  specific  tokens  such  as  • ;* 
and  including  legal  starting  tokens  for  general 
grammatical  statements) 


. 
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2)  unstacking  of  the  state  stack  until  a  read  state  which 
can  legally  read  the  above  hard  token  is  found 
(empty  stack  implies  a  repetition  of  (1  or  other 
recovery  mecanisms) . 
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6  -  Conclusions 


The  LALR  (k)  parser  generator  described  has  been  designed 
and  implemented  to  produce  efficient  parsers  both  in  the  space 
and  time  domains.  Empirical  proof  of  its  efficiency  has  been 
submitted  in  the  attached  appendix.  The  only  real  questions  left 
are  "can  it  be  made  more  efficient  ?",  and  "can  it  be  upgraded 
into  an  LR  (k)  parser  generator  ?".  The  answers  of  course  are 
"yes".  Clearly,  a  more  intelligent  state  comparing  algorithm  can 
be  devised  to  find  more  complicated  subsets  of  states  (implying 
more  space  saving) .  Another  algorithm  can  be  devised  to  delete 
states  with  identical  transitions  (which  provide  no  necessary 
left-context).  Lastly,  a  state-splitting  algorithm  can  be 
inserted  to  handle  the  elusive  LR(k)  but  not  LALR  (k)  condition. 


■ 
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2_~_Appendix 


7.0  The_Inf or ma 1_K nut h^Ea rley_A Igor  it h m 

This  very  simple  algorithm  can  best  be  explained  by 
example.  Consider  the  following  grammar: 

G  ->  1-  A  -|  t  ( 1) 

B  ->  X  #  (2) 

A  ->  BX  #  (3) 

A  ->  (A)  #(4) 

Since  the  program  goal  is  G,  the  start  state  is  denoted  by 
the  initial  configuration  composed  of  a  single  marked 
production: 

State  1  G  ->  1-  A  -  |  #  (1) 

where  "  |-"  (the  tagged  symbol)  is  underlined  to  show  that 
it  is  a  symbol  which  can  be  read  while  in  this  state. 

A  state  is  identified  by  a  configuration  and  represented  by 
a  set  of  transitions  (composed  of  transition  symbols  and 
destination  states) .  The  tagged  symbols  in  the  configuration 
become  the  transition  symbols  of  the  state.  The  "J-"  will 
therefore  be  a  transition  symbol. 

A  configuration  is  composed  of  an  initial  configuration  and 
completers.  Completers  are  formed  by  adding  on  to  the  initial 
configuration  newly  "started"  productions  which  are  subgoals  of 
a  symbol  being  read. 
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In  State  1,  we  have  no  such  subgoals  and  hence  the 

completion  of  the  initial  configuration  is  the  same 

configuration. 

Therefore,  in  state  1,  symbol  u | - ”  can  be  read  implying  a 
transition  to  a  new  state.  Before  naming  it  state  2,  its  initial 
configuration  must  be  created  and  then  completed. 

The  initial  configuration  is 
G  ->  |-  A  -|  #  (1 ) 

Since  A  is  a  subgoal,  additions  must  be  made  to  this 
configuration;  namely 

A  ->  B  X  #  ( 3) 

A  ->  _{  A)  #  (4) 

In  the  process,  we  have  generated  another  subgoal,  namely 
B.  Therefore,  it  too  must  be  added;  that  is 

B  ->  X  #  (2) 

The  completed  configuration  is  unique  (has  not  been 
previously  created) ;  hence  we  name  it  uniquely  as  state  2. 

State  2  G  ->  |-  A  -|  #  ( 1 )  ,  A  ->  B  X  #  (3)  ,  A  ->  i  A)  #(4), 
B  ->  X  #  (2) 

State  1  is  now  completely  generated  and  can  be  represented 
as 

State  1  G  ->  J_-  A  -  |  #  (1) 
j -  go  to  2 


. 
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State  2  on  the  other  hand  must  yet  be  generated  by  filling 
in  the  following 

A  go  to  ? 

B  go  to  ? 

(  go  to  ? 

X  go  to  ? 

Consider  the  first  symbol,  namely  A;  its  destination  state 
must  be  identified  by  the  following  configuration 
G  ->  I-  A  -i  #  (1 ) 

which  is  in  itself  completed.  Call  it  state  3,  since  it 
also  is  unique.  Therefore  reading  an  A  while  in  state  2  implies 
a  transition  to  state  3  defined  as 

State  3  G  ->  |-  A  —  J_  #  ( 1 ) 

Consider  next  transition  symbol  B  in  state  2.  Its 
destination  state  will  be  a  unique  state  (call  it  4)  defined  by 

State  4  A  ->  B  X  #  (3 ) 

The  destination  state  of  transition  symbol  " ("  in  state  2 
will  have  the  initial  configuration 

A  ->  (  A  )  #  (4) 

The  completers  to  this  configuration  are 

A  ->  B  X  #  ( 3) 

A  ->  i  A)  #  (4) 

B  ->  X  #  (2) 

The  completed  configuration  is  thus  unique  and  defines 


state  5 


' 


■ 


. 


* 

. 


State  5 
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State  5 

A  ->  (  A  )  #  (4)  ,  A  ->  B  X  #  (3)  ,  A  ->  ±  A)  #  (4)  , 

B  ->  X  #  (2) 

The  last  transition  symbol  in  state  2,  namely  X,  will  have 


the  following 

destination  state: 

State  6 

B  ->  X  #J21 

*  (2) 

#  (2)  means  apply  production  2, 

The  above  state  (state  6)  is  already  generated  and  merely 
states  that  a  production  must  be  applied. 

State  2  at  this  point  is  completely  generated  and  appears 


as 

State  2 

G  ~>  1-  A  -  j  #  (1 )  ,  A  ->  B  X  #  (3)  ,  A  ->  J  A)  #  (4)  , 

B  *>  X  #  (2) 

A  go  to  3 

B  go  to  4 

(  go  to  5 

X  go  to  6 

State  3  becomes 


State  3 

G  S  -1  #(1) 

- 1  go  to  7 

State  7 

G  ->  |-  A  -|  #111 

#  (1) 

Similarly  state  4  becomes 
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State  4  A  ->  B  X  #  ( 3) 

X  go  to  8 

State  8  A  ->  B  X  #_£ 3)_ 

#<3) 

State  5  on  the  other  hand  is  more  interesting.  Re-copying, 
it  looks  as  follows 

State  5  A  ->  (  A  )  #  (4)  ,  A  ->  B  X  #  (3)  ,  A  ->  _(  A)  #  (4)  , 

B  ->  X  #  (2) 

A  go  to  ? 

B  go  to  ? 

(  go  to  ? 

X  go  to  ? 

The  destination  state  of  transition  symbol  A  is  identified 
by  the  configuration  A  ->  (A  ]_  #  (4)  which  is  unique  and 

complete.  Therefore  call  it  state  9  defined  as: 

State  9  A  ->  (A  #  (4) 

The  destination  state  of  transition  symbol  B  will  also 
define  a  new  configuration  A  ->  B  X  # (3 )  and  called  state  10 

State  10  A  ->  B  X  #{3) 

Similarly  "  (w  will  have  a  destination  state  defined  by 
configuration  A  ->  (  A  )  #  (4 )  with  completers  A  ->  B  X  # (3)  ,  A 

->  |  A)  #{4),  B  ->  X  #  (2)  .  This  is  state  5. 

Lastly  X  will  have  a  destination  state  defined  by 
configuration  B  ->  X  #_[2]L  which  is  state  6. 
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State  5  when  completely  generated  appears  as  follows 


State  5 

A  ->  (  A  )  #(4),  A  ->  B  X  f  (3)  ,  A  ->  J  A)  t  (4)  , 

B  ->  X  #  (2) 

A  go  to  9 

B  go  to  10 

(  go  to  5 

X  go  to  6 

All  that 

remains  now  is  to  generate  states  9  and  10. 

State  9 

A  ->  (A  l  #  (4) 

)  go  to  11 

State  11 

A  ->  (A)  #141 

♦  (4) 

State  10 

A  ->  B  X  #  (3) 

X  go  to  12 

State  12  A  ->  BX  #J31 

#<3) 

In  summary,  the  machine  is  completely  defined  as  follows. 


State  1 

g  ->  i:  a  -  |  #(i) 

|-  go  to  2 

State  2 

G  ->  |-  A  -|  #(1),  A  ->  B  X  #  (3)  ,  A  ->  1  A)  #  (4)  , 

B  ->  X  #  (2) 

A  go  to  3 

B  go  to  4 

(  go  to  5 

X  go  to  6 

. 


. 

■ 


f 


State  3  G  ->  |-  A  -J.  #  (1) 


State  4 


State  5 


State  6 


State  7 


State  8 


State  9 


State  10 


State  1 1 


State  12 


-|  go  to  7 
A  ->  B  X  #  { 3) 

X  go  to  8 

A  ->  (  A  )  #{4),  A  ->  B  X  #  (3)  ,  A  ->  1  A)  1(4) 

B  ->  X  #  (2) 

A  go  to  9 
B  go  to  10 
(  go  to  5 
X  go  to  6 
B  ->  X  #J2)_ 

#  (2) 

G  ->  1  -  A  -  j  #1U 

#(1) 

A  ->  BX  #131 
*  (3) 

A  ->  (A  1  #  (4) 

)  go  to  11 
A  ->  B  X  #  ( 3) 

X  go  to  1 1 
a  ->  (A)  mi 

#{4) 

A  ->  BX  #111 


#(3) 


' 
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7.1  Table  Organization 

To  insure  that  identical  or  top-most  subset  states  are  not 
duplicated,  a  system  of  indirect  addressing  is  used.  INDEX  1  and 
INDEX2  respectively  points  to  other  arrays  containing  the  read, 
lookahead  or  apply  production  states  information,  and  contains 
the  number  of  elements  in  each  of  the  above  states. 

In  order  to  keep  the  width  of  the  array  small,  bit  (8)  as 
opposed  to  bit  (16)  numbers  if  possible,  the  information  related 
to  the  three  types  of  states  mentioned  above  was  partitioned 
into  three  sets  of  corresponding  arrays;  namely  READ1 
(containing  read  terminals)  and  READ2  (containing  the 
destination  state) ,  L00K1  (containing  look  ahead  terminals)  and 
LOOK 2  (containing  the  destination  state),  and  finally  APPLY  1 
(containing  states  that  can  be  on  top  of  the  state  stack)  and 
APPLY2  (containing  the  destination  state) .  Clearly  the  indexes 
will  be  smaller  than  if  they  were  all  merged  into  one  big  array. 

Push  states  were  built-in  to  the  index  arrays,  INDEX1 
(containing  the  destination  state)  and  INDEX2  (containing  the 
push  state).  Since  the  pushed  state  is  a  read  state  and  the 
wgo  to”  state  is  an  apply  production  state,  then  it  is  clear 
that  the  first  item  is  smaller  than  the  second  and  should  be  put 
in  INDEX2  (which  so  far  contains  only  small  numbers) ,  in  an 
effort  to  keep  the  width  of  INDEX2  down  to  bit  (8) . 

The  ”In  Any  Case”  condition  is  implemented  in  both 
lookahead  and  apply  production  states  by  having  the 


- 
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corresponding  left  array  pair  member 
code,  "0”  to  be  specific.  INDEX 2  for 
not  used.  Pull  numbers  for  the  appl 
therefore  stored  in  the  corresponding 


contain  an 
these  states 
y  production 
INDEX2  array. 


"In  Any  Case" 
are  therefore 
states  are 


INDEX  1 


INDEX2 


State  Type 


Read 

READ  1,2  ptr 

#  of  elements 
in  state 

Look 

LOOK  1,2  ptr 

Unused 

Push 

Destin.  State 

Value  to  Push 

Apply 

APPLY  1,2  ptr 

pull  # 

READ  1  READ2 


read 

corresp . 

Terminals 

destination 

state 

LOOK  1  LOOK  2 


lookahead 

corresp. 

Terminals 

destination 

state 

APPLY  1  APPLY2 


top  of 

corresp. 

state 

destination 

stack 

state 

The  last  element  in  L00K1  and  APPLY1  is  an  "In  any  case" 
condition,  coded  as  "0".  A  destination  state  coded  as  "0"  for 
APPLY2  implies  "STOP"  (goal  symbol  reached). 


. 
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7.2  Simultaneous  Lef t  and  Right-  Recursion 

That  simultaneous  left  and  right  recursion  is  ambiguous  can 
be  justified  by  the  following  example.  Consider  the  two 
productions : 

B  ->  AB  #  (1) 

B  ->  BC  #  (2) 

where  A,C  could  be  any  string  of  one  or  more  terminals,  or  non¬ 
terminals.  Then  there  exists  two  distinct  parses  for  the  string 
ABC,  namely: 


ABC  ABC 


In  other  words,  having  read  A  and  B  and  possibly  looked 
ahead  at  C,  we  cannot  tell  if  we  should  apply  the  productions  in 
the  orders  #  {1)  •  #  (2)  or  #  (2)  ,  #{1)  (both  being  legal). 

The  same  situation  develops  for  A  ->  ABA  if  string  ABABA  is 
examined  with  the  middle  A  in  mind  and  so  on. 


. 


' 
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7.3  Empirica  1_S  j3ace_0bserva  tions 

Initial  decisigns:^  We  chose  to  produce  tables  for  the  IBM 
System/360,  in  the  form  of  initialized  declarations  in  the  XPL 
language  [8,9];  this  imposed  the  constraint  that  storage  be 
allocated  in  multiples  of  8  bits,  but  should  not  drastically 
affect  our  results.  More  important,  rather  than  representing  the 
possible  state  transitions  by  a  matrix,  we  chose  to  store  them 
as  tables  of  ordered  pairs;  this  seems  more  compact  encoding, 
but  has  potential  speed  implications  (discussed  later).  Finally, 
we  devoted  some  effort  in  the  constructor  to  combining  obviously 
redundant  entries  in  the  tables. 

The  basis  for  comparison:  We  initially  chose  the  mixed- 
strategy  precedence  (MSP)  constructor  [8/9]  as  the  basis  for 
comparison.  It  seemed  a  natural  choice  for  a  number  of  reasons: 
we  have  an  implementation,  the  XPL  System  [8,9],  in  current  use; 
we  are  throughly  convinced  of  its  usefulness  and  efficiency;  it 
was  the  system  we  were  considering  replacing. 

After  obtaining  preliminary  comparisons  with  MSP,  we 
decided  to  expand  our  experiment  to  include  another  precedence 
method.  Wirth-Weber  simple  precedence  (WWSP)  [10]  is  widely 
known  and  historically  important.  Additionally,  we  could 
accurately  estimate  table  size  without  actually  implementing  or 
running  the  constructor,  merely  by  computing  the  storage 
required  for  the  precedence  matrix  plus  the  productions.  It  was 
therefore  easy  to  include  WWSP  in  our  comparison. 


. 


. 
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The  sample  grammars:  For  each  precedence  method  we  took  a 
grammar  of  a  programming  language  which  was  developed 
specifically  for  use  with  that  method:  XPL  [8,9]  for  MSP  and 
EULER-  [10]  for  WWSP.  We  actually  evaluated  two  versions  of  the 
EULEH  grammar:  one  (exactly  as  published  in  [10]  contains  20 
productions  defining  <nuraber>;  the  other  (EULER  -  <number>)  has 
these  20  productions  removed  (since  this  level  of  detail  is 
normally  relegated  to  the  compiler’s  lexical  scanner).  Any  bias 
in  these  selections  presumably  favors  precedence  methods. 
Additionally,  we  included  an  ALGOL  60  grammar,  which  is 
presumably  not  biased  towards  any  particular  parsing  method,  (we 
made  the  modifications  described  in  [7]  to  remove  syntactic 
ambiguities,  but  did  not  otherwise  alter  the  grammar) . 

Although  this  is  not  a  large  sample,  we  feel  that  it  is 
typical  of  the  grammars  our  TWS  will  be  required  to  handle  (once 
their  syntactic  ambiguities  have  been  removed) . 


# 


The  results 


Grammar 

Vocabulary 

Terminal 

size 

Nonterminal 

Number  of 
Productions 

XPL 

42 

49 

109 

EULER 

78 

44 

120 

EULER- 

<number> 

65 

39 

100 

ALGOL  60 

62 

82 

173 

Grammar 

MSP 

WWSP 

LALR 

bytes 

bytes 

bytes 

state 

XPL 

3274 

* 

1250 

234 

EULER 

3922 

4321 

1606 

223 

EULER- 

3017 

3204 

1276 

192 

<number> 

ALGOL  60 

>6800** 

>6100* 

2821 

376 

*  Not  a  WWSP  grammar 


**  Not  a  MSP  grammar 


• 
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These  results  speak  for  themselves.  The  LALR  tables  are 
significantly  smaller  than  either  set  of  precedence  tables. 
These  sizes  may  be  related  to  actual  compilers  by  noting  that 
the  compiler  for  XPL  (XCOM)  [9]  requires  105,000  bytes  of 
program  and  data,  and  that  the  MSP  and  LALR  parsers,  including 
error  recovery,  require  1868  and  1416  bytes  respectively.  Thus, 
the  MSP  parser  and  tables  constitute  5%  of  the  space  of  XCOM, 
while  the  LALR  parser  and  tables  would  be  under  3%. 
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7.4  Em^irical  Speed  0 bser vations 

The  large  size  of  the  precedence  tables  in  the  previous 
section  is  principally  due  to  the  use  of  precedence  matrices 
which  grow  guadratically  with  the  vocabulary  size.  Had  we 
represented  the  state  transitions  by  matrix,  the  LALK  tables 
would  also  have  been  large.  Our  representation  saved  substantial 
space,  but  at  some  penalty  in  speed,  by  requiring  table  search, 
rather  than  simple  indexing.  We  wondered  if  the  resulting  parser 
would  be  impractically  slow. 

We  could  have  substituted  the  new  parser  into  an  existing 
compiler  (e.g.,  XCOM)  and  measured  the  difference  in  compile 
time;  however,  parsing  is  a  small  fraction  of  compilation,  and 
we  were  unsure  how  reliably  the  difference  could  be  observed. 
Instead,  we  abstracted  just  the  lexical  scan  and  parse  routines 
from  XCOM .  To  separate  card  reading  and  scanning  time  from 
parsing  time,  we  had  the  program  prescan  an  entire  program  at  a 
time,  storing  numerical  codes  for  the  tokens  in  a  large  array, 
and  then  parse  from  the  array.  The  results  (on  the  IBM  360/44) 
for  three  different  XPL  programs  are  given  in  the  following 


table ; 


. 


- 


. 


63 


Size 

Humber  of 

MSP 

LALB 

Program 

Cards  Tokens 

Reductions 

seconds 

seconds 

Corapactif y 

77 

439 

1,262 

0.84 

0.52 

LCS 

3,322 

17,369 

58,707 

28.86 

15.88 

XCOM 

4,241 

24 ,390 

66 ,108 

45.  35 

25.  1 1 

SIMPAC 

5,577 

24,990 

92,245 

46.11 

25.38 

DIAL 

6,405 

32,136 

1 16,803 

58.24 

32.65 

DOSYS 

7,291 

29,334 

81,581 

55.58 

30.49 

Again,  the  results  indicate  substantially  greater 
efficiency  for  the  L ALB  parser.  The  fraction  of  the  compiler’s 
time  required  for  parsing  drops  from  19%  to  11%. 

It  is  not  immediately  obvious  why  MSP,  with  its  fast 
indexing  into  the  precedence  matrix,  is  slower  than  LALB  with 
its  relatively  slow  table  lookups.  The  reason  is  rooted  in  one 
of  the  basic  differences  between  precedence  and  LB  (k)  methods. 
The  precedence  matrix  can  only  be  used  to  locate  the  leftmost 
reducible  substring  in  the  stack;  a  table  of  right  parts  must 
still  be  searched  to  find  the  applicable  production.  With  LB (k) 
methods,  however,  the  same  function  that  locates  the  reducible 
substring  simultaneously  indicates  the  applicable  production, 
without  an  additional  table  search.  This  saving  apparently 
outweighs  the  cost  of  the  searching  the  state  table. 


■ 
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